Artificial intelligence intra-operative surgical guidance system and method of use

ABSTRACT

The inventive subject matter is directed to a computing platform configured to execute one or more automated artificial intelligence models, wherein the one or more automated artificial intelligence models includes a neural network model, wherein the one or more automated artificial intelligence models are trained on a plurality of radiographic images from a data layer to detect a plurality of anatomical structures or a plurality of hardware, wherein at least one anatomical structure is a pelvic teardrop and a symphysis pubis joint; detecting at a plurality of anatomical structures in a radiographic image of a subject, wherein the plurality of anatomical structures are detected by the computing platform by the step of classifying the radiographic image with reference to a subject good side radiographic image; and constructing a graphical representation of data, wherein the graphical representation is a subject specific functional pelvis grid; the subject specific functional pelvis grid generated based upon the anatomical structures detected by the computing platform in the radiographic image. Various types of functional grids can be generated based on the situation detected.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of prior application Ser. No.17/668,319 filed Feb. 9, 2022, which is a continuation in part of U.S.Ser. No. 16/916,876 filed Jun. 30, 2020, and also claims the benefit ofPCT/US2019/050745 filed Sep. 12, 2019, under 35 USC Sec. 371 and U.S.provisional patent application No. 62/730,112 filed Sep. 12, 2018, under35 U.S.C. Sec. 119(e) (hereby incorporated by reference in theirentirety).

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

None.

Reference to a “Sequence Listing,” a table, or a computer programlisting appendix submitted on a compact disc and anincorporation-by-reference of the material on the compact disc: None.

FIELD OF THE INVENTION

The subject of this invention is an artificial intelligenceintraoperative surgical guidance system in joint replacements, spine,trauma fracture reductions and deformity correction, and implantplacement/alignment. A method is provided for analyzing subject imagedata, calculating surgical decision risks and autonomously providingrecommended pathways or actions that support the decision-making processof a surgeon to predict optimized implant and subject outcomes (ex.implant guidance, fracture reduction, anatomical alignment) by agraphical user interface.

BACKGROUND OF THE INVENTION

Many of the radiographic parameters essential to total hip arthroplasty(THA) model performance, such as wear and stability, can be assessedintraoperatively with fluoroscopy. However even with intraoperativefluoroscopic guidance, the placement of an implant or the reduction of abone fragment can still not be as close as desired by the surgeon. Forexample, mal positioning of the acetabular model during hip arthroplastycan lead to problems. For the acetabular implant to be inserted in theproper position relative to the pelvis during hip arthroplasty requiresthat the surgeon know the position of the patient's pelvis duringsurgery. Unfortunately, the position of the patient's pelvis varieswidely during surgery and from patient to patient. During traumasurgery, proper fracture management, especially in the case of anintra-articular fracture, requires a surgeon to reduce the bone fragmentoptimally with respect to the original anatomy in order to: provide theanatomical with joint the best chance to rehabilitate properly; minimizefurther long-term damage; and, if possible, to regain its normalfunction. Unfortunately, in a fracture scenario, the original anatomicalposition of these bone fragments has been compromised and their naturalrelationship with the correct anatomy is uncertain and requires thesurgeon to use his/her best judgment in order to promote a successfulrepair and subsequent positive outcome. During a surgery, a surgeon isrequired to make real-time decisions that can be further complicated bythe fact that there are multiple decisions needing to be made at thesame time. At any given time, there can be a need for a decision made ona fracture reduction guidance for example and simultaneously a decisionrequired on implant placement and an error at any stage will increasethe potential for a sub-optimal outcome and potential surgical failure.Unfortunately, most of these problems are only diagnosed and detectedpostoperatively and oftentimes lead to revision surgery. These risks andpatterns need to be identified in real-time during the surgical ormedical event. As surgeons and medical professionals must often relysolely on themselves to identify hazards and risks or make decisions oncritical factors in, and surrounding, a surgical event, a need existsfor a system and method that can provide intraoperative automatedintelligence guided surgical and medical situational awareness supportand guidance.

SUMMARY OF THE INVENTION

This summary describes several embodiments of the presently disclosedsubject matter and, in many cases, lists variations and permutations ofthese embodiments. This summary is merely exemplary of the numerous andvaried embodiments. Mention of one or more representative features of agiven embodiment is likewise exemplary. Such an embodiment can typicallyexist with or without the feature(s) mentioned; likewise, those featurescan be applied to other embodiments of the presently disclosed subjectmatter, whether listed in this summary or not. To avoid excessiverepetition, this summary does not list or suggest all combinations ofsuch features.

The novel subject matter includes an artificial intelligenceintra-operative surgical guidance system made of: a computing platformconfigured to execute one or more automated artificial intelligencemodels, wherein the one or more automated artificial intelligence modelsare trained on data from a data layer, wherein the data layer includesat least surgical images, to calculate intra-operative surgical decisionrisks, and to provide an intra-operative surgical guidance to a user.More specifically, the computing platform is configured to provide anintra-operative visual display to the user showing surgical guidance.The computing platform is trained to show intra-operative surgicaldecision risks by applying an at least one classification algorithm.

The novel subject matter includes: a computing platform configured toexecute one or more automated artificial intelligence models, whereinthe one or more automated artificial intelligence models includes aneural network model, wherein the one or more automated artificialintelligence models are trained on a plurality of radiographic imagesfrom a data layer to detect a plurality of anatomical structures or aplurality of hardware, wherein at least one anatomical structure isselected from the group consisting of: a pelvic teardrop and a symphysispubis joint; detecting at a plurality of anatomical structures in aradiographic image of a subject, wherein the plurality of anatomicalstructures are detected by the computing platform by the step ofclassifying the radiographic image with reference to a subject good sideradiographic image; and constructing a graphical representation of data,wherein the graphical representation is a subject specific functionalpelvis grid; the subject specific functional pelvis grid generated basedupon the anatomical structures detected by the computing platform in theradiographic image.

In another embodiment, the inventive subject matter includes: anartificial intelligence assisted total hip arthroplasty involving thesteps of: providing a computing platform comprised of an at least oneimage processing algorithm for the classification of a plurality ofintra-operative medical images, the computing platform configured toexecute one or more automated artificial intelligence models, whereinthe one or more automated artificial intelligence models including aneural network model wherein the one or more automated artificialintelligence models are trained on data from a data layer to identify aplurality of orthopedic structures and a plurality of hardware;receiving a preoperative radiographic image of a subject; detecting aplurality of anatomical structures in the preoperative radiographicimage of the subject; generating a subject specific functional pelvisgrid from the plurality of anatomical structures detected in thepreoperative radiographic image of the subject, receiving anintraoperative anteroposterior pelvis radiographic image of the subject;identifying an object selected from the group consisting of: an at leastone anatomical landmark and an at least one hardware in theintraoperative anteroposterior pelvis radiographic image, whereby thecomputing platform performs the step of: selecting a situation specificgrid selected from the group consisting of: functional pelvis grid,spinopelvic grid, level pelvis grid, reference grid, neck cut grid,reamer depth grid, center of rotation grid, cup grid, leg length andoffset grid, and femur abduction grid. In another embodiment, theinventive subject matter includes an artificial intelligence basedintra-operative surgical guidance system comprising: a non-transitorycomputer-readable storage medium encoded with computer-readableinstructions which form a software module and a processor to process theinstructions, wherein the software module is comprised of a data layer,an algorithm layer and an application layer, wherein the artificialintelligence based intra-operative surgical guidance system is trainedto detect a plurality of anatomical structure, wherein the computingsoftware is configured to detect a plurality of anatomical structures ina radiographic image of a subject, wherein the plurality of anatomicalstructures are detected by the computing platform by the step ofclassifying the radiographic image with reference to a subject good sideradiographic image; and construct a graphical representation of data,wherein the graphical representation is a subject specific functionalpelvis grid; the subject specific functional pelvis grid generated basedupon the plurality of anatomical structures detected by the computingplatform in the radiographic image. More specifically, the methodincludes: constructing a graphical representation of data, wherein thegraphical representation is an anatomy map of an entire anatomicalregion; the anatomy map generated based upon the plurality of anatomicalstructures detected by the computing platform; quantify measurementsbetween the plurality of anatomical structures in the anatomy map;receiving required output from a user; and calculating the requiredoutput based on a vector or vertical distances measured from structuresin said plurality of anatomical structures in the anatomy map.

BRIEF DESCRIPTION OF THE SEVERAL IMAGES OF THE DRAWINGS

The drawings show the apparatus and method of use according to anexample form of the present invention. The invention description refersto the accompanying drawings:

FIG. 1A is a diagram of the system for automated intraoperative surgicalguidance.

FIG. 1B shows an exemplary view of a head-up display image of thesystem.

FIG. 2A is a diagram of the computing platform.

FIG. 2B is a diagram of an artificial intelligence computing system.

FIG. 3A. is a schematic illustration of Deep learning as applied toautomated intraoperative surgical guidance.

FIG. 3B is a schematic illustration of reinforcement learning automatedintraoperative surgical guidance.

FIGS. 4A and 4B are block diagrams of the software modules.

FIG. 5A is an overview of preoperative workflow of the presentinvention.

FIG. 5B is an overview of intraoperative workflow of the presentinvention.

FIG. 5C is an image of a postoperative workflow of the presentinvention.

FIG. 6 is a preoperative image.

FIG. 7A is a preoperative image shown with procedure specificapplication relevant information as the input with the resultant tasksand actions.

FIG. 7B is the output of the anatomical position model showing a guidetemplate of good pose estimation.

FIG. 8A is a graphical user interface showing image with grid templateshowing the anatomical outlines of what is a good pose.

FIG. 8B shows the best image pose guidance process.

FIG. 8C shows the output of the use of a reference image.

FIG. 9A is a graphical user interface showing an image with anatomicalfeatures defined.

FIG. 9B shows user inputs, task and actions.

FIG. 10 is a graphical user interface showing anatomical measurementgrid positioned on an image.

FIG. 11A is a graphical user interface showing an image of the affectedside.

FIG. 11B shows user inputs, task and actions.

FIG. 12A is a graphical user interface showing a display of acceptedimage classification.

FIG. 12B shows user inputs, task and actions relating to the formulationof a treatment plan.

FIG. 13A is a graphical user interface showing ghosting.

FIG. 13B shows the data flow in ghosting.

FIG. 14A is a graphical user interface showing grid similarity withmatch confidence display.

FIG. 14B shows the output of a graphical user interface showingconfirmation of good side for further use.

FIG. 14C shows the good side acceptance.

FIG. 15A is a graphical user interface showing an image of the good-sideoverlay with grid alignment and measurements.

FIG. 15B is a graphical user interface showing an image of the good-sideoverlay with grid alignment and measurements for an ankle.

FIG. 15C is a graphical user interface showing an image of the good-sideoverlay with grid alignment and measurements for a nail.

FIG. 16 is a representation of statistical inference of 3D models and anexample of the use of representation of statistical inference of 3Dmodels.

FIG. 17A shows an image of anatomy alignment and fracture reductionguidance.

FIG. 17B shows user inputs, task and actions related to the datasets.

FIG. 18A is a graphical user interface instrument guidance.

FIG. 18B shows user inputs, task and actions related to the datasets.

FIG. 18C shows various outputs.

FIG. 18D shows the predictive and contra-side matched ideal entry point.

FIG. 19A is a graphical user interface showing instrument guidance andor virtual implant placement.

FIG. 19B is a graphical user interface showing instrument guidance andor virtual implant placement of lag screw placement.

FIG. 20A is the workflow of an artificial intelligence assisted totalhip arthroplasty (THA).

FIG. 20B is the workflow of an artificial intelligence landmark andhardware detection.

FIG. 20C is the workflow of an artificial intelligence assisted griddetection.

FIG. 21A shows images of an anatomy map generated by artificialintelligence assistance for a total hip arthroplasty (THA).

FIG. 21B shows images of a virtual functional pelvis grid generated byartificial intelligence assistance for a total hip arthroplasty (THA).

FIG. 22 shows images of intra-operative image calibration.

FIG. 23A show intraoperative image of an anteroposterior pelvis withinstruments and an implant.

FIG. 23B show intraoperative image of an anteroposterior pelvis withinstruments and an implant.

FIG. 23C shows intraoperative image of an anteroposterior pelvis withinstruments and multiple implants.

FIG. 24 shows intraoperative image of the anteroposterior pelvis implant(cup) and stem (trial).

FIG. 25 shows intraoperative images of an anteroposterior hip and no cupor stem.

FIG. 26A shows an intraoperative image of an AP pelvis along with animplant cup and stem.

FIG. 26B show an intraoperative image of an AP pelvis along with animplant cup and stem.

FIG. 26C show an intraoperative image of an AP pelvis along with animplant cup and stem.

FIG. 27A shows use of a grid template in a hip arthroplasty (THA).

FIG. 27B shows use of a grid template providing leg length and hipoffset data to a user in a graphical format.

FIG. 28 shows a functional pelvis grid.

FIG. 29 shows a neck cut grid.

FIG. 30 shows a reamer depth grid.

FIG. 31 shows a center of rotation grid.

FIG. 32 shows a cup abduction grid.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

The present invention can be understood more readily by reference to thefollowing detailed description of the invention. It is to be understoodthat this invention is not limited to the specific devices, methods,conditions or parameters described herein, and that the terminology usedherein is for describing embodiments by way of example only and is notintended to be limiting of the claimed invention. Also, as used in thespecification including the appended claims, the singular forms “a,”“an,” and “the” include the plural, and reference to a numerical valueincludes at least that value, unless the context clearly dictatesotherwise. Ranges can be expressed herein as from “about” or“approximately” one value and/or to “about” or “approximately” anothervalue. When such a range is expressed, another embodiment includes fromthe one value and/or to the other value. Similarly, when values areexpressed as approximations, by use of the antecedent “about,” it willbe understood that the value forms another embodiment. All combinationsof method or process steps as used herein can be performed in any order,unless otherwise specified or clearly implied to the contrary by thecontext in which the referenced combination is made. These and otheraspects, features and advantages of the invention will be understoodwith reference to the detailed description herein and will be realizedby means of the various elements and combinations particularly pointedout in the appended claims. It is to be understood that both theforegoing general description and the following detailed description ofthe invention are exemplary and explanatory of preferred embodiments ofthe inventions and are not restrictive of the invention as claimed.Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs.

The following system and method relate to a computing platform having agraphical user interface for displaying subject image data and applydata science techniques such as machine and deep learning to: calculatesurgical decision risks, to predict a problem and provide guidance inreal-time situations. The system autonomously displays recommendedactions through a display such as graphical user interface to provide anoptimized implant and subject outcome by calculating the probability ofa successful procedural outcome (ex. Implant guidance, fracturereduction, anatomical alignment). The inventive subject matter isdirected to an artificial intelligence intra-operative surgical guidancesystem and method of use. The system in its most basic form included: acomputer executing one or more automated artificial intelligence modelstrained on at least intra-operative surgical images, to calculatesurgical decision risks, and to provide an intra-operative surgicalguidance, and a visual display configured to provide the intra-operativesurgical guidance to a user.

Artificial Intelligence is the ability of machines to perform tasks thatare characteristics of human intelligence. Machine learning is a way ofachieving Artificial Intelligence. AI is the ability of machines tocarry out tasks in an intelligent way. Machine learning is anapplication of Artificial Intelligence that involves a data analysis toautomatically build analytical models. Machine learning operates on thepremise that computers learn statistical and deterministicclassification or prediction models from data; the computers and theirmodels adapt independently as more data is inputted to the computingsystem. Misinterpretation of data can lead to mistakes and a failedoutcome. Artificial Intelligence can integrate and infer from a muchlarger and smarter dataset than any human can discerning patterns andfeatures that are difficult to appreciate from a human perspective. Thisbecomes particularly relevant in the alignment of anatomy and correctplacement of implants. The system analyzes and interprets theinformation and provides guidance based upon a correlation to a knownset of patterns and inference from novel sets of data. The artificialintelligence intra-operative surgical guidance system is made of acomputer executing one or more automated artificial intelligence modelstrained on data layer datasets collections to calculate surgicaldecision risks and provide intra-operative surgical guidance; and adisplay configured to provide visual guidance to a user.

Now referring to FIGS. 1A and 1B, the artificial intelligenceintra-operative surgical guidance system 1 includes an imaging system110. The imaging system 110 receives subject image data such as images120 (radiographic, ultrasound, CT, MRI, 3D, terahertz) of a subject'sanatomy. Exemplary medical images that can be analyzed forintraoperative surgical guidance can include a radiographic image suchas an image generated by portable fluoroscopy machine called a C-arm. Insome embodiments, the computing platform 100 can be configured toperform one or more aspects associated with automated intraoperativesurgical guidance in medical images. For example, computing platform 100and/or a related module can receive an inter-operative surgical image,e.g., a fluoroscopic image of the knee.

The artificial intelligence intra-operative surgical guidance system 1includes an input of a series of x-ray or fluoroscopic images of aselected surgical site, a computing platform 100 to process the surgicalimages and an overlay of a virtual, augmented, or holographicdimensioned grid, with an output to an electronic display device 150.The electronic display device 150 provides a displayed composite imageand graphical user interface 151. The graphical user interface 151 isconfigured to: allow manipulation of a dimensioned grid 200 by a user155, such as a surgeon, physician assistant, surgical scrub nurse,imaging assistant and support personnel. The computing platform 100 isconfigured to synchronize with a sensor 130 to (a) provideintraoperative anatomical (for example, bone) or implant positionalinformation; and (b) provide postoperative anatomical or implant orexternal alignment and correction device information to an ArtificialIntelligence Engine for guidance.

The computing platform 100 is configured to synchronize with a surgicalfacilitator 160 such as a robot or a haptic feedback device 162 toprovide the same predictive guidance as described throughout as anenabler for robotic surgery. The computing platform 100 is configured tosynchronize with an intelligence guided trackable capable of creatingaugmented grids or avatars of implants, instruments or anatomy toprovide the same predictive guidance as described throughout as anenabler for intelligence guided artificial reality trackable navigation.

The system components include an input of a series of x-ray orfluoroscopic images of a selected surgical site, a dynamic surgicalguidance system 1 to process the surgical images and an overlay of avirtual, augmented, or holographic dimensioned grid 200 with an image120, with an input device to provide manipulation of the dimensionedgrid 200 by a user 155, such as a surgeon. In one embodiment, theelectronic display device 150 is an electronic display device, such as acomputer monitor, or a heads-up display, such as GLASS (Google). Inanother embodiment, the electronic display screen 150 is a video fpvgoggle. An out-put to an electronic display device 150 is provided forthe user 155 to image the overlay of the series of images and thedimensioned grid 200.

The augmented reality or holographic dimensioned grid 200 can bemanipulated by the user 155 by looking at anatomic structures, the shownon the electronic display device 150 that will facilitate locking on thecorrect alignment/placement of surgical device. The artificialintelligence intra-operative surgical guidance system 1 allows the user155 to see critical work information right in their field-of-image usinga see-through visual display and then interact with it using familiargestures, voice commands, and motion tracking. The data can be stored indata storage. The artificial intelligence intra-operative surgicalguidance system 1 allows the user 155 to see critical work informationin their field-of-image using a see-through visual display device 150and then interact with it using familiar gestures, voice commands, andmotion tracking through a graphical user interface 151 such as by anaugmented reality controller. The graphical user interface 151, such asaugmented reality or holographic dimensioned grid, can be manipulated bythe user 155 by looking at anatomic structures, then shown on theelectronic display device 150 that will facilitate locking on thecorrect alignment/placement of surgical device.

FIGS. 2A & 2B are diagrams illustrating an exemplary artificialintelligence surgical guidance system 1 including a computing platform100 for dynamic surgical guidance according to an embodiment of thesubject matter described herein. A computer platform is a system thatincludes a hardware device and an operating system that an application,program or process runs upon.

The subject matter described herein can be implemented in software incombination with hardware and/or firmware. For example, the subjectmatter described herein can be implemented in software executed by atleast one processor 101. In one exemplary implementation, the subjectmatter described herein can be implemented using a computer readablemedium having stored thereon computer executable instructions that whenexecuted by the processor of a computer platform to perform the steps.Exemplary computer readable media suitable for implementing the subjectmatter described herein include non-transitory devices, such as diskmemory devices, chip memory devices, programmable logic devices, andapplication specific integrated circuits. In addition, a computerreadable medium that implements the subject matter described herein canbe located on a single device or computing platform or can bedistributed across multiple devices or computing platforms. As usedherein, the terms' “function” or “module” refer to hardware, firmware,or software in combination with hardware and/or firmware forimplementing features described herein.

The computing platform 100 includes at least one processor 101 andmemory 102. The computing device can invoke/request one or more serversfrom the Cloud Computing Environment 98 other clinical metadata can beefficiently retrieved from at least one server from the CloudEnvironment if it is sorted in only one server or from separate serversif the dataset was sorted partially in different servers; some outcomesderived from the AI engine can be directly sent and sorted in one ormore servers in the Cloud platform (privacy is preserved).

The computing platform 100 analyzes an image for risk factors that theuser cannot see due to their human inability to interpret anoverwhelming amount of information at any specific moment. If theimplant placement and the alignment does not match this data pattern, itwill create an awareness in this specific situation and provide a hazardalert to the user. Essentially, identifying and predicting problemsahead of the user encountering them. This can lead to avoidance ofcomplications and prevention of errors. The computing platform 100includes a plurality of software modules 103 to receive and processmedical image data, including modules for image distortion correction,image feature detection, image annotation and segmentation, image toimage registration, three-dimensional estimation from two-dimensionalimages, medical image visualization, and one or more surgical guidancemodules that use artificial intelligence models to classify images aspredictive of optimal or suboptimal surgical outcomes. The term dynamicor dynamically means automated artificial intelligence and can includevarious artificial intelligence models such as for example: machinelearning, deep learning, reinforcement learning or any other strategiesto dynamically learn. In a trauma event, such as fracture reduction ordeformity correction, or in an arthroplasty event such as hip or kneeanatomical alignment or bone cut guidance, or in the event of a spineprocedure with implant alignment correction, or in a sports medicineevent with ACL reconstruction alignment, these surgical procedurespecific datasets coupled with domain knowledge that are useful to anevent can be accessed. They will be used to interpret critical failuremode factors of an implant or anatomical alignment and combined willprovide the user with a Failure Risk Score with the output to the useras a confidence percentage recommendation of a suboptimal or optimalperformance metric. This will be presented to the user in the form ofintelligent predictors and scores to support decisions encountered in areal time event.

The software module 103 includes a plurality of layers. The data layer105 is made of a collection of data from various managed distributeddata collection networks. This collection of data represents theknowledge that is necessary to address specific tasks. Data layer(detailed in FIG. 2B) is the collection of data, which is the input foralgorithm layers 106, and it contains data sets from different systemswhich makes Data layer 105 a rich source information needed for thedifferent deployment tasks. These data layers 105 include surgical imagedata and related outcomes, medical advisor knowledge base and sensordata.

The algorithm layer 106 includes computer-implemented methods speciallydesigned to target application layer using inputs provided in Data Layer105. The algorithm layer 106 can also be referred to as an AI engine.The algorithm layer 106 includes various image processing algorithmswhich use different machine learning techniques (engineering features)and artificial intelligence techniques (hierarchical learningfeatures/learned representation of the features). All algorithms aredesigned to solve different tasks such as image enhancement, edgedetection, segmentation, registration, etc. With the help of Algorithmlayer 105, these tasks will be performed automatically, which willcontribute to the understating of the high-level complexity of medicaldata and also to the understanding of dependencies among the dataprovided in Data Layer. The algorithm layer 106 also includes learningalgorithms such as statistical models and prediction models.Representative examples include image quality scoring algorithm, Deeplearning algorithm, Machine Learning based algorithms, and imageregistration algorithms.

The computing platform 100 is configured to execute one or moreautomated artificial intelligence models. The one or more automatedartificial intelligence models are trained on data from the data layer105. The data layer 105 includes at least a plurality of surgicalimages. The artificial intelligence intra-operative surgical guidancesystem 1 includes a computing platform trained to calculateintra-operative surgical decision risks by applying an at least oneclassifier. More specifically the computing platform is trained toperform the classification of intra-operative medical images of implantsfixation into discrete categories that are predictive of surgicaloutcomes, for instance, optimal and sub-optimal. The automatedartificial intelligence models are trained to calculate intra-operativesurgical decision risks and to provide an intra-operative surgicalguidance, and a visual display configured to provide the intra-operativesurgical guidance to a user. The application layer 107 includes but isnot limited to: clinical decision support, surgical guidance, riskfactor and other post processing actions such as image interpretationand visual display.

Now referring to FIGS. 3A&B example of automated artificial intelligencemodels are shown. The computing platform 100 is configured to executeone or more automated artificial intelligence models. These one or moreautomated artificial intelligence models are trained on data from a datalayer 105.

These automated artificial intelligence models include: Deep Learning,machine learning and reinforcement learning based techniques. Forexample, a Convolutional Neural Network (CNN) is trained usingannotated/labeled images which include good and bad images to learnlocal image features linked to low-resolution, presence ofnoise/artifact, contrast/lighting conditions, etc. The CNN model usesthe learning features to make predictions about a new image. The CNNmodel can include a number of conventional layers and a number ofpooling layers which proceed to subsampling (or down sampling) of eachfeature map while retaining the most informative feature. The stack ofthe layers can include various Conventional Kernels of size N×M; N and Mare positive integers and stand respectively for Kernel width andheight.

FIG. 3A illustrates a deep learning model architecture. The AI deep CCNarchitecture shown in FIG. 3A is for the classification for surgicaloutcomes (optimal vs suboptimal) with a score quantifying thisclassification using Convolutional Neural Networks. This architecturecomprises several (deep) layers involving linear and non-linearlearnable operators which enable building high-level information makingthe process of construction of discriminative information automated. Thefirst input layer of the deep learning network learns how to reconstructthe original dataset which is the collection of data layer 105. Thesubsequent hidden layers learn how to reconstruct the probabilitydistributions of the activations of the previous layer. The number ofthe hidden layers define the depth level of the deep learningarchitecture. The output layer of a neural network is tied to theoverall task.

As illustrated in the figure FIG. 3A, the classification CNN provides anoutput probability score for surgical outcome. This score quantifies thequality of positioning of an implant and/or bone alignment. CNNhyperparameters including the number of layers as well as the filtersizes were derived empirically upon testing the performance of thedesigned network on data collection from Data Layer. The CNNarchitecture is adjustable in a way to provide high sensitivitydetection for the positioning of the implant.

FIG. 3B is a schematic illustration of multi-scale reinforcement leaning(RL) as applied to the task of intraoperative screw insertion. This typeof reinforcement learning can intraoperatively illustrate insertiontrajectory and pedicle screw trajectories.

Now referring to FIGS. 4A&B, the software is organized into modules andeach module has at least one function block as shown in this figure. Thenon-transitory computer-readable storage medium is coupled to amicroprocessor, wherein the non-transitory computer-readable storagemedium is encoded with computer-readable instructions that implementfunctionalities of the following modules, wherein the computer-readableinstructions are executed by a microprocessor. The computing platform100 of the artificial intelligence intra-operative surgical guidancesystem 1 can include the following modules. Module 5 is made of an imagequality scoring algorithm to assess the quality of an acquired medicalimage for its intended use. The image quality scoring algorithm is animage processing algorithm that is based on Machine Learning or Deeplearning from a good and bad medical image training dataset for aspecific application. For Machine Learning based algorithms, imagequality score of as given image is computed based on quality metricswhich quantify the level of accuracy in which a weighted combination oftechnical image factors (e.g., brightness, sharpness, etc.) relate tohow clearly and accurately the image captures the original anatomicalstructures of an image. These weighed combinations of factors are knownpredictors of optimal or sub-optimal outcomes or performance measures.Examples: “adequacy of reduction” (FIG. 28B). The weighted combinationof technical factors is a parametrized combination of key elements whichquantify how good the image is. It can be seen as an indicator ofrelevancy of the image and determines if the acquired image issufficient to work with or not. In this invention, it is used to definethe quality metric/image score

For Deep learning-based techniques, a Convolutional Neural Network (CNN)is trained using annotated/labeled images which include good and badimages to learn local image features linked to low-resolution, presenceof noise/artifact, contrast/lighting conditions, etc. The CNN model usesthe learning features to predict, for a new image, its image qualityscore.

As can be seen in FIG. 3A, the CNN model can include a number ofconventional layers and a number of pooling layers which proceed tosubsampling or down sampling of each feature map while retaining themost informative feature. The stack of the layers can include variousConventional Kernels of size N×M; N and M are positive integers andstand respectively for Kernel width and height. Module 5 maximizes theperformance of further computer vision and image processing tasks.Module 5 can also include a grid-based pose guide to assist the user inacquisition of a better image as appropriate to the application.

Module 6 includes one or more algorithms to detect and correct fordistortion inherent in medical imaging modalities, for example thefluoroscopic distortion inherent in intraoperative C-arm imaging.

Module 7 is an image annotation module that includes image processingalgorithms or advanced Deep learning-based techniques for detectinganatomical structures in a medical image and identifying contours orboundaries of anatomical objects in a medical image, such as bone orsoft tissue boundaries. Anatomical Landmark detection stands for theidentification of key elements of an anatomical body part thatpotentially have a high level of similarity with the same anatomicalbody part of other patients. The Deep learning algorithm encompassesvarious conventional layers, and its final output layer providesself-driven data, including, but not limited to, the system coordinatesof important points in the image. In the current invention, landmarkdetection can be also applied to determine some key positions ofanatomical parts in the body, for example, left/right of the femur, andleft/right of the shoulder. The Deep Neural Network output is theannotated positions of these anatomical parts. In this case, the Deeplearning algorithm uses a training dataset which needs to meet somerequirements: the first landmark in the first image used in the trainingmust be consistent across different images in the training dataset.Identifying contours of anatomical objects refers to providing an edgemap consisting of rich hierarchical features of an image whilepreserving anatomical structure boundaries using Deep learningtechniques. A variety of highly configurable Deep learning architectureswith an optimized hyperparameters tuning are used to help with solvingspecific tasks. The trained Conventional Neural Network in oneembodiment includes tuned hyperparameters stored in one or manyprocessor-readable storage mediums and/or in the Cloud ComputingEnvironment 98.

Module 8 is a preoperative image database including computer algorithmsand data structures for storage and retrieval of preoperative medicalimages, including any metadata associated with these images and theability to query those metadata. Preoperative images can includemultiple imaging modalities such as X-ray, fluoroscopy, ultrasound,computed tomography, terahertz imaging, or magnetic resonance imagingand can include imagery of the nonoperative, or contralateral, side of apatient's anatomy.

Module 9 is the Image Registration which includes one or more imageregistration algorithms.

Module 10 is composed of computer algorithms and data structures for thereconstruction and fitting of three-dimensional (3D) statistical modelsof anatomical shape to intraoperative two-dimensional orthree-dimensional image data. Module 11 is composed of image processingalgorithms and data structures for composing multiple medical images,image annotations, and alignment grids into image-based visualizationsfor surgical guidance.

Module 12 is an Artificial Intelligence Engine that is composed of imageprocessing algorithms based on Machine and/or Deep learning techniquesfor the classification of intraoperative medical images of reduction andalignment procedures into discrete categories that are predictive ofdiffering surgical outcomes, such as suboptimal or optimal outcomes.Classifications produced by Module 12 can also include an associatedscore that indicates a statistical likelihood of the classification andis derived from the model underlying the image classifier algorithm,i.e., a classifier.

Module 13 is an Artificial Intelligence Engine that is made of imageprocessing algorithms which uses Machine Learning or Deep learningmethods for the classification of intraoperative medical images ofimplant fixation procedures into discrete categories that are predictiveof differing surgical outcomes, such as suboptimal or optimal.Classifications produced by Module 13 can also include an associatedscore that indicates a statistical likelihood of the classification andis derived from the model underlying the image classifier algorithm.

Module 14 is a postoperative image database made of computer algorithmsand data structures for storage and retrieval of postoperative medicalimages, including any metadata associated with these images and theability to query those metadata. Postoperative images can include imagesacquired during routine follow-up clinic visits or surgical revisions.

Module 15 is an Artificial Intelligence Engine that is made of imageprocessing algorithms for the classification of a time series ofpostoperative medical images into discrete categories that arepredictive of differing surgical outcomes, such as suboptimal or optimaloutcomes. Classifications produced by Module 15 can also include anassociated score that indicates a statistical likelihood of theclassification and is derived from the model underlying the imageclassifier algorithm.

Module 16 is a fracture identification and reduction module with accessto an AO/OTA Classification Dataset interprets the image and makes aclassification of the bone, bone section, type and group of thefracture.

Now referring to FIG. 2B, the computing platform 100, which includes oneor more Artificial Intelligence (AI) Engines, including FIG. 4A, Modules12, 13, and 15, and information from a series of datasets. Here deepneural networks and other image classifiers are trained to analyze andinterpret visual features in one or more images to anticipate problemsand predict outcomes in a surgery or in a postoperative follow-upperiod. Training relies on one or more medical image datasets withassociated known outcomes data. A trained neural network in this contextcan thus be thought of as a predictive model that produces a surgicaloutcome classification from an input set of medical image features.

The outcome classification is typically also accompanied by astatistical likelihood that the classification is correct. Together, theclassification and its likelihood can be thought of as an outcomeprediction and a confidence level of that prediction, respectively. Inthe case of a suboptimal outcome prediction, we can consider theconfidence level to be a Failure Risk Score for a suboptimal outcome.The classification and Failure Risk Score can thus be used by thesurgeon to support decisions that lead to optimal outcomes and avoidsuboptimal outcomes. Any number of classical machine learning approachescan be used, as well as more modern Deep learning networks [LeCun, Yann,Yoshua Bengio, and Geoffrey Hinton. “Deep learning.” nature 521.7553(2015): 436], such as Convolutional Neural Networks [e.g., Lawrence,Steve, et al. “Face recognition: A convolutional neural-networkapproach.” IEEE transactions on neural networks 8.1 (1997): 98-113.] Asurgical outcomes classifier with a confidence score can also beconstructed using any number of methods in multivariate statistics,including a Cox proportional hazards model or other commonregression-based time-to-failure models constructed from clinicalresearch data. In the case of a classifier constructed using amultivariate statistical model, the inputs include at least in partfeature sets derived from the medical image data. For instance, in orderto identify surgical outcomes using “non-image” datasets, for examplediagnosis reports derived from datasets (e.g. “Dataset Outcomes SurgicalVariables” in FIG. 4B), Natural Language Processing (NPL) can be used toprocess clinical text data.

The systems and methods describe uses for the artificial intelligentplatform, such as the ability to read and interpret subject image data,and calculate surgical decision risks, and provide the end user with aconfidence score of the probability of an optimal outcome and predictorof performance metrics for implants and surgical factors. This occurs bydynamically updating, by the computing platform, the composite imagewith the at least one surgical guidance.

The computing platform 100, which includes an artificial intelligenceengine, utilizes and analyzes the information from the datasets. Theseinformation sets have been analyzed and structured and based upon thespecific surgical application can include: procedural medical imagedatasets, such as intraoperative fluoroscopic images and pre- andpostoperative x-ray, MRI or computed tomography data; an AO/OTADanis-Weber fracture classification dataset; Lauge-Hansen classificationsystem dataset; implant 3D CAD model datasets, biomechanical testingsuch as Von Mises Stresses failure modes datasets; medical image featuredatasets and learned models for anatomical feature tracking; best-posegrid datasets; fracture reduction image datasets with associatedoutcomes data; other surgical outcomes datasets: peer-reviewedliterature and clinical studies datasets; known predictors andindicators of complications datasets; 3D statistical models of humananatomy datasets; other medical image datasets; an expert physiciandomain knowledge datasets; bone quality index datasets; failure riskscore datasets; subject HER information data; and outcomes surgicalvariables datasets such as trauma outcomes data, arthroplasty outcomesscoring data, ACL outcome rating scales, and spine scoring systems.

In addition to these surgical and procedure specific datasets,information from subject health records such as comorbidity data, thepresence of deformity, and bone quality index scores can be accessed.These datasets are configured to include information that willpotentially have an impact on the outcome of the procedure. The datasetsare used to interpret critical failure mode factors of an implant oranatomical alignment and when used to train an outcomes classifier foran Artificial Intelligence Engine provides the user with a prediction ofoptimal or suboptimal outcome and an associated Failure Risk Score. TheAI engine include multiple CNNs based classifiers which can be selectedusing the specific dataset (one or more dataset, most importantlyuncorrelated data that make the CNN learn new relevant features) fromData Layer for solving a well-defined task, for example, determine theposition of implants, etc.

The information from independent datasets can be accessed at any giventime, or alternatively a situation during the event can require theinput from various datasets simultaneously. In this situationinformation from the relevant datasets will be selected for inclusion inthe Artificial Intelligence (AI) Engine in the form of multiple trainedclassifiers, each with a weighted contribution to the final surgicaloutcome prediction. In this case, Machine and/or Deep learningtechniques are intended to identify relevant image features from inputspace of these datasets and the AI Engine seeks an individual customizedsoftware solution to a specific task, for example a decision regardingimplant positioning or surgical guidance, and datasets involved to solvethat task. This multiple prediction model utilizes information fromdatasets that have a relationship from the perspective of sharinguncorrelated or partially correlated predictors of a specific outcome.The AI Engine can further weight the outcome prediction data based uponthe relative level of criticality regarding performance or failure. Themodel outputs decision support and outcome predictions for example theprobability of a successful and optimal long-term outcome.

The computing platform 100 is configured to synchronize with a ComputerAssisted Surgery (CAS) system to provide the same predictive guidance asdescribed throughout as an enabler for computer assisted surgery. Thedynamic surgical guidance system 1 described herein, has the capabilityto provide predictive guidance or act as an enabler for subjectspecific, or custom, matched block guided technology. For example, thepresent invention can be applicable to other musculoskeletalapplications such as arthroplasty surgery for hip, knee, ankle andshoulder as well as trauma surgery for musculoskeletal repair and forspine applications. Typical applications include hip, knee, shoulder,elbow, and ankle arthroplasty, trauma fractures and limb deformitycorrection, spine, and sports medicine procedures such asfemoroacetabular impingement/(FAI)/Periacetabular Osteotomy (PAO). Theartificial intelligence intra-operative surgical guidance system 1 isconfigured to implement a method including the steps of: obtainingsubject image data; dynamically displaying the subject image data on agraphical user interface; selecting an anatomical structure within thesubject image data and mapping a grid template to the anatomicalstructure to provide a registered image data; providing an artificialintelligence engine and at least one dataset configured to generatesurgical guidance; providing as a data output, the registered imagedata, to the artificial intelligence engine to generate at least onesurgical guidance; and dynamically updating, by the computing platform,the composite image of the registered image data with the at least onesurgical guidance. The surgical guidance is related to: deformitycorrection, an anatomy alignment, a fracture reduction and an anatomyreduction. The process of surgical guidance will be discussed in thefollowing section for these applications. The method further includesthe step of generating a trackable location and orientation guided bythe grid template. These steps will be more fully described as they areimplemented in FIGS. 5A-31 .

Now referring to FIGS. 4A,& 5A-5C, an over image of the use of anartificial intelligence engine and the data sets applied to registeredpre-, intra-, and postoperative images yields a graphical user interfacefor use in reduction and alignment and implant fixation procedures. Thepreoperative workflow is provided. The workflow proceeds as follows. Theimaging system 110 of the artificial intelligence intra-operativesurgical guidance system 1 receives subject image data such as one ormore preoperative images and computes an image quality score using ImageQuality Scoring and Pose Guide Module 5. The user is presented with achoice to either accept or reject the image based on the image qualityscore and pose guide guidance. If the image is rejected, the operatortries again to acquire an acceptable image. If accepted, distortion inthe image is detected and corrected using the Distortion CorrectionModule 6. The image is then annotated with anatomical structures andimage segmentations using the Image Annotation Module 7. Images are thenstored for later use in the intraoperative and postoperative workflowsvia the Preoperative Image Database Module 8. The process is thenrepeated to acquire any number of images necessary for later referencein the intraoperative and postoperative workflows.

Now referring to FIGS. 4A & 5B, the intraoperative workflow is provided.The process proceeds as follows. The artificial intelligenceintra-operative surgical guidance system 1 receives one or morepreoperative images and computes an image quality score using ImageQuality Scoring and Pose guide Module 5. The user is presented with achoice to either accept or reject the image based on the image qualityscore and pose guide guidance. If the image is rejected, the operatortries again to acquire an acceptable image. If accepted, distortion inthe image is detected and corrected using the Distortion CorrectionModule (6). The image is then annotated with anatomical structures andimage segmentations using the Image Annotation Module 7. The artificialintelligence intra-operative surgical guidance system 1 then registersthe image to the best matching corresponding image in Preoperative ImageDatabase module 8 and computes a matching score using the ImageRegistration Module 9. The user accepts or rejects the image andregistration based on the registration match and quality score. Ifaccepted, three-dimensional anatomical shape information can be computedusing the 3D Shape Modeling Module 10 followed by a registration(mapping) of an alignment grid to annotated image using the ImageRegistration Module 9.

The step of registration is the process of transforming images ofpreoperative of nonoperative side (the fixed image, f(x),) andintraoperative of the operative side (the current moving image, m(x),)to a common coordinate system so that corresponding pixels representhomologous biological points. This means recovering the transform, T(x),which maps points in f(x) to m(x). This is accomplished by the steps of:(1) define the transformation model, (2) determine the similaritymetrics describing the objective function to be minimized, and (3) theoptimization algorithm that solves the minimization problem. Theeffective alignment of these images will allow the surgeon to highlightdifferent characteristics and therefore establish a better comparison ofthese images. It should be noted that the images that are registered donot have to be imaged from the same modality; it can be MRI to CT or CTto CT, and so on.

The computing platform 100 of the artificial intelligenceintra-operative surgical guidance system 1 produces a composite image orimages for display to the user using any combination of the currentacquired image, the aligned preoperative image, the registered alignmentgrid using the Image Composition Module 11. Here different processes arefollowed depending on the type of procedure. For reduction & alignment,the system computes an outcome classification and Failure Risk Scoreusing the Reduction and Alignment Outcomes Prediction Module 12. Forimplant fixation, the system computes an outcome classification andFailure Risk Score using the Implant Fixation Outcomes Prediction Module13.

The artificial intelligence intra-operative surgical guidance system 1then annotates the displayed composite image and graphical userinterface with the outcome classification and Failure Risk Score, alongwith any surgical guidance information. Surgical guidance directives canthen be communicated to a surgical facilitator 160 such as a hapticfeedback device, a robot, a trackable guide such as tracked Implant orobject, a cutting block, a computer assisted surgery device, IoT deviceand a mixed reality device.

Now referring to FIGS. 4A & 5C, the postoperative workflow is provided.The process proceeds as follows. The artificial intelligenceintra-operative surgical guidance system 1 receives one or morepostoperative images and computes an image quality score using ImageQuality Scoring and Pose guide Module (5). The user is presented with achoice to either accept or reject the image based on the image qualityscore and pose guide guidance. If the image is rejected, the operatortries again to acquire an acceptable image. If accepted, distortion inthe image is detected and corrected using the Distortion CorrectionModule 6. The image is then annotated with anatomical structures andimage segmentations using the Image Annotation Module 7. The artificialintelligence intra-operative surgical guidance system 1 then registersthe image to all preceding time series images in the Postoperative ImageDatabase 14 and computes matching scores using the Image RegistrationModule 9. The user accepts or rejects the image and registration basedon the registration match and quality score. If accepted,three-dimensional anatomical shape information can be computed using the3D Shape Modeling Module 10, followed by a registration (mapping) of analignment grid to annotated image using the Image Registration Module 9.

The artificial intelligence intra-operative surgical guidance system 1produces a composite image or images for display to the user using anycombination of the current acquired image, the aligned preoperativeimage, the registered alignment grid using the Image Composition Module11. The system then computes an outcome classification and Failure RiskScore using the Postoperative Outcomes Prediction Module 13. Theartificial intelligence intra-operative surgical guidance system 1 thenannotates the displayed composite image and graphical user interfacewith the outcome classification and Failure Risk Score, along with anyguidance information.

Now referring to FIG. 6 , the subject is prepared and positioned for amedical or surgical event in a standard manner as indicated for thespecific procedure, for example, joint replacement, orthopedic trauma,deformity correction, sports medicine, and spine. The preoperative image115 or data is imported and is shown as FIG. 6 . The preoperative image115 shows a grid template 200 super imposed over the subject'sanatomical image.

A grid template 200 has a plurality of dimensioned radio-opaque lines,e.g., 230 relating to surgical variables. The portion of the gridtemplate 200 that is not opaque is radiolucent. The grid template 200can include any shape or pattern of geometric nature or text toreference angles, length positioning or targeting. The grid template 200can be a single line, a geometrical pattern, number, letter or a complexpattern of multiple lines and geometries that correspond to surgicalvariables. The grid patterns can be predesigned or constructedintraoperatively in real-time based upon the surgeon's knowledge ofanatomy and clinical experience including interpretation of morphometricliterature and studies identifying key relationships and dimensionsbetween anatomical structures and its application in supporting goodsurgical technique as it relates to specific procedures. With respect toa digital dimensioned grid, this form of the grid template 200 isgenerated by the application software.

The subject is prepared and positioned for a medical or surgical eventin a standard manner as indicated for the specific procedure, forexample, joint replacement, orthopedic trauma, deformity correction,sports medicine, and spine. The procedure specific information for therespective application is extracted from the preoperative image 115 ordata and mapped into live intraoperative images 120. Mapping is definedas computing a best-fit image transformation from the preoperative tothe intraoperative image space. The transformation is made of thecomposition of a deformation field and an affine or rigidtransformation. The best fit transformation is computed using a varietyof established methods, including gradient descent on mutualinformation, cross-correlation, or the identification of correspondingspecific anatomical structures in preoperative 115 and intraoperativeimages 120. See, e.g., U.S. Pat. No. 9,610,134 specifically incorporatedby reference in its entirety.

Now referring to FIGS. 7A, a new image 120 of the unaffected anatomy isacquired and transmitted (can be wirelessly) to the computing platform100. At the beginning of the procedure, the user will use the softwareof the computing platform 100 to assist with anatomical positioning,namely the Image QS and Pose module as shown in FIG. 4A. The computingplatform 100 identifies structures on the intraoperative image 120 anddetermine an optimal pose for the image to be taken. Structures areidentified based on classifiers learned from the medical image datasets.Outcome classifiers can take the form of a deep neural network, atemplate matching algorithm, or a rule-based classifier or decisiontree.

Now referring to FIG. 7B, the is computing platform 100 provides areal-time guide template 250 of good pose estimation. The guide template250 is a guide for the user to acquire a best-fit opportunity for theartificial intelligence intra-operative surgical guidance system 1 tomatch subsequent new images 120 with the dataset of optimal-pose-imagesas shown on an electronic display device 150. For example, in an ankle,once a pose is selected, the lateral image can be segmented intospecific anatomical features and a guide template 250 mapped to thesefeatures—tibia, fibula, and talus.

FIGS. 8A-C shows the overlay mapping of images and pose guide template250 to provide a pose-guide image 260. Once the user has acquired thepreferred or correct image pose, then that image and/or guide template250 can be used as the guidance pose-guide image 260 for the remainderof the procedure. The guidance pose-guide image 260 can be that of acontralateral or unaffected side of the body, or a best-fit image from adataset, or a statistical shape model, or a geometric virtual grid.

The user takes subsequent images until satisfied it matched the guidancepose-guide image 260 required or a threshold is detected. The correctpose can be acquired in two ways, 1) by adjusting position of anatomy(subject), or 2) by adjusting pose/angle of imaging equipment (exC-arm). This process can be manually instructed by the user orautonomously performed by the software module of the computing platform100. The computing platform 100 attempts to determine whether thematched image 270 is a good match for one of the images 120 in thepreoperative image database. Here the computing platform 100 uses theImage Registration Module 9 as shown in FIG. 3A.

This process involves a multi-scale image-based registration metric thatcan be quickly applied to the image pairs. If a match above a thresholdis detected, the computing platform 100 attempts to automaticallyidentify relevant anatomical structures in the new image using any ofthe techniques for image landmark classifiers. Landmark information andoptionally other image information is used to compute a transformation Tof the new image to the coordinate space of the preoperative image.

Now referring to FIG. 8B, auto landmarking and smart image registrationoccur at this time whereby the computing platform 100 attempts toautomatically identify relevant anatomical structures, such as posteriordistal fibula 271, in the matched image 270. Any of the aforementionedtechniques for landmark classifiers can be used. In addition, image nailapproaches based on edge and feature recognition, landmarkidentification, statistical atlas registration, and deformable modelscan be used to segment relevant areas of anatomy from the image, such asthe talus 272. In these images, the matched image 270 is a template thatacts a pose guide 250 in this situation.

Now referring to FIG. 8C the best image pose guidance good side withimage registration module 9 and pose guide module 5 in addition toexpert domain knowledge dataset is accessed as shown. Once the best poseimage is accepted, the image registration module 9 identifies theapplication specific feature(s) 280, for example the talus in an anklefracture procedure. The output is an automatic identification andselection of the desired anatomical features and a grid template 200positioned relative to these features 280 displaying image and gridalignment.

Now referring to FIGS. 4A and 5 , in the image registration module 9, auser such as a surgeon, selects at least one anatomical landmark in theanatomical image on the graphical user interface 151. Anatomicallandmark selection can be accomplished by a various methods includingbut not limited to: auto-segmentation where the software of thecomputing platform 100 uses feature/pattern recognition process toauto-detect known and targeted anatomical structures; use of a remoteinfrared device, such as a gyro mouse; voice command; air gestures; gaze(surgeon uses gaze and direction of eye or head motion to controltargeting) or touching the visualization screen at the selectedanatomical structures.

In one illustrative embedment, the surgeon inputs the selection of theanatomical structures to the workstation manually or using a variety ofinput devices such as, an infrared wand or an augmented reality device.The application software of the computing platform 100 registers a gridtemplate 200 with the selected anatomical structures. The methodincludes the step of registering an anatomical image to a grid template200 by selecting at least one anatomical landmark to provide a gridtemplate 200 with at least one grid indicator 280. A grid indicator 280is an anatomical feature defined and located on an image that correlateswith a known position on a virtual grid template 200 for purposes ofregistration. If needed a registration procedure is used to eitherunwarp the image or warp the digital grid indicators according to theimage warping.

The software of the computing platform 100 identifies and recognizescalibration points that are radiopaque in the image. These are of knowndimensioned geometries. A grouping of these points is a distortioncalibration array. The distortion calibration array is placed on theimage intensifier or in the field of image of any imaging system so thatthe known distortion calibration array lines/points are identified whenan image is taken and captured. These known patterns are saved for usein the distortion adaptation/correction process. The distortioncalibration array is removed from visualization on the display medium tonot obscure and clutter the image with unnecessary lines/points. Adistortion calibration array can be made a series of lines or pointsthat are placed to support the distortion adaptation of the gridtemplate 200. The distortion calibration array points or lines areradiopaque so that the distortion process can calculate the location ofthese points/lines relative to the anatomy and quantify the amount ofdistortion during each image taken. Once these points/lines areidentified and used in the distortion process, there is another processthat removes the visualization of these points/lines from the anatomicalimage so that they are not obstructing the surgeon's image when he orshe sees the grid template 200 and the anatomical image.

In one embodiment, the registration process involves manually orautomatically detecting grid structures (such as grid lineintersections, points, and line segments) on the grid template 200superimposed on the anatomical image and then aligning those structuresvia an Affine Registration and a deformation field with correspondingstructures on a distortion calibration array of known geometry, which isa represented digitally. The method includes the step of deforming thecalibrated dimensioned grid to correct for the distortion of theanatomical image to generate a deformed calibrated dimensioned gridimage. Known radiopaque lines/points (from distortion calibration array)are used to provide a measure of EM distortion in each anatomical image.The distortion is quantified and then the software of the computingplatform 100 generated virtual grid is adapted to match the distortedanatomy in each anatomical image.

The distortion calibration array is of non-uniform design, such that theselected anatomical structures are clustered more densely in regions ofinterest to the surgeon, in order that the deformation correction can beestimated with greater precision in those regions. The deformationestimation proceeds as follows: once selected anatomical structures havebeen identified (either manually or automatically) on the array image,an Affine Transformation that produces the best mapping betweencorresponding selected anatomical structures from the grid template 200to the array image is computed. Following transformation of the gridpoints by the Affine Transformation, which adjusts the structures fortranslation, rotation, and scaling with respect to the array imagestructures in the Deformation Field (which is the residual differencebetween transformed grid points and the array image points) is modeledusing Thin-Plate Splines or any other suitable radial basis functions.Parameters of the Thin-Plate Splines or radial basis functions areestimated by solving a linear system of equations. U.S. patentapplication Ser. No. 15/383,975 (hereby specifically incorporated byreference). The array image becomes the reference image or thecalibrated image.

Once the deformation field has been computed, the dimensioned grid isadapted in real-time intraoperatively to fit the subject anatomy, thusproducing a distorted grid indicator, such as lines curving that can beused to match or fit the musculoskeletal anatomy or the shape of theimplant. The deformation of the grid indicators is then applied inreal-time by first applying the Affine Transformation and then warpingthe grid indicators along the Deformation Field. A grid pattern basedupon the anatomical points that was defined and targeted in landmarkidentification is generated. The software of the computing platform 100is configured to compute the amount of distortion in each image and itquantifies this amount relative to the anatomical image and thendisplays the calculated grid/Image relationship displaying an image ofthe subject's anatomy with the quantitatively distorted dimensioned gridimage. These deformed grids are tracked in real time with each new imagetaken. The deformed grid can be positioned relative to anatomy, implant,and fractures auto or manually by the user such as a surgeon. Numerousequations and formulas are used within the algorithms to calculate:measurements, differences, angles, grid and implant positions, fracturedeviations to determine at least one measurement of surgical variablesinvolving the implant or trauma.

In auto-segmentation, the at least one anatomical landmark selected bythe surgeon is automatically selected for each successive anatomicalimage. Auto-segmentation allows a surgeon to work more rapidly.Auto-segmentation is accomplished through a combination of one or moreof the following techniques: intensity thresholding; feature detectionon the intensity edges in the image, including shape detection via theHough Transform or other methods; feature detection followed byregistration of a 2D or 3D anatomical atlas with predefined landmarkpositions.

Now referring to FIGS. 9A&B and 10, an intraoperative image withanatomical features defined is shown as a graphical user interface. Agrid template 200 is mapped to the identified anatomical object orimplant feature to form a matched grid map image 290. In this image theobject is a lateral talus 280. The computing platform 100 is configuredto auto track features and calculate new coordinate positions andintegrated grid map of anatomy, objects, and implants. The graphicaluser interface is i) locked in auto tracking of subsequent images and,ii) auto integrate and position an alignment grid. Using the graphicaluser interface, an alignment grid 350 is positioned in the matchedimage. The user adjustments are accepted and the computing platform 100is configured to provide a matched grid map 290.

Now referring to FIGS. 11A&B, an image 121 of the affected side isobtained. The affected side could show a deformed, injured or diseasedlimb. The process previously described for the good side is repeated forthe affected side. In this process, the best pose and auto identifyanatomical features, objects and implants occurs by matching to the goodside of the subject. The inputs to the process are: affected side image,good side pose guide data and feature identification of the data set.The task conducted by the computing platform 100 is to estimate the bestpose match with the good side and auto map the feature and the grid map.The action required is calculate the best pose, auto identify featuresand calculate the matched grid map. The output is the good side(contralateral), and afforded side are matched with the alignment gridis positioned on the image.

Now referring to FIGS. 12A&B, a fracture identification and reductionmodule 16 with access to an AO/OTA Classification Dataset interprets theimage and makes a classification of the bone, bone section, type andgroup of the fracture. As shown in FIG. 12A as a graphical userinterface, the user then can accept the classification. In an exemplaryembodiment, in the event that the procedure involves a fracture of thebone, the affected side image is analyzed for a fracture and the type offracture classified 45 according to the AO/OTA classification system.For example, an ankle fracture selection classified as metaphysealcomplex (43-A3). Next if the classification is accepted, then the taskand actions include providing the treatment option 47.

As shown in FIGS. 12 B&C, a treatment option 47 is provided based on theclassification selected. Treatment option 47 is shown as a graphicaluser interface. Here a trauma plate implant 49 is shown for use in thisfracture situation. This is accomplished by the computing platform 100configured to acknowledge the selection of fracture type from therelevant dataset, and algorithmic modules are accessed for a treatmentplan analysis and suggestion for the user 155. The treatment option 47will include a determination of the recommended implants for fixation ofthis type of fracture classification, for example an implant plate withspecific screw configurations and selections 49. Automatic fractureidentification can be accomplished using a classifier trained withvarious machine learning approaches, including deep convolutional neuralnetworks or rule-based classifiers.

More specifically, the CNN model is trained on datasets which includeimages with one or more fractures and other images without fractures.Then, the CNN model determines whether there is a fracture or not andalso localizes the region of interest which contains the identifiedfracture and/or the abnormality. Precise identification of the fracturearea in the image is critical information required to support theclassification, in addition to providing evidence regarding the fixationprocess for the type of fracture. In practice, the input image isprocessed by a CNN model (representative architecture is illustrated inFIG. 3A, which includes various conventional and max-pooling layers, inorder to produce feature maps. At the last layer, the network ismodified to target a candidate ‘fracture region of interest’ on theimage based upon feature map information giving the location and size ofthe region of interest. Candidate regions, including all fracturelocation and suspected abnormality detections, marked by the CNN modelare then shown.

Now referring to FIGS. 13A&B, a guidance pose-guide image 260 can alsobe used as the good-side reference image to be used throughout theprocedure as a template image for the similarity evaluation and mappingmodule whereby the anatomy of the good, or unaffected, side is matchedwith the image of the affected side anatomy 121. This is accomplished bythe computing platform 100 configured to use image analysis andsegmentation techniques to autonomously perform a similarity evaluationto identify and register bony anatomical features, in real-time, of theaffected and contralateral images. This technique is identified asghosting. In ghosting, an overlay image 50 is obtained by overlaying thegood-side image 120 versus the bad side image 121 with reference to theguidance pose-guide image 260. Matching of the operative andcontralateral-side images is accomplished by computing a best-fit imagetransformation. The transformation can include the composition of adeformation field and an affine or rigid transformation. The best fittransformation can be computed using a variety of established methods,including gradient descent on mutual information, cross-correlation, orthe identification of corresponding specific anatomical structures inpreoperative and intraoperative images.

More specifically, as shown in FIGS. 5A-5C, the ghosting procedureinvolves the steps of: 1) Before the surgery, operating room personnelacquires preoperative image(s) of the nonoperative side of the patient'sor subject's anatomy. For example, in an ankle fracture case, standardanterior-posterior, lateral, and mortise images of the uninjured anklewould be taken. These images might also be ordered by a surgeon to beacquired by radiology using standard x-ray before the case and thenloaded onto our device before the surgery. 2) The nonoperative-sideimages acquired in step 1 are processed by the computing platform 100 toidentify key structures that will later be used for image registrationwith the operative side. Additionally, these images may be corrected fordistortion and may have a grid template overlaid. 3) During thereduction phase of the surgery, images of the operative side areacquired that the surgeon uses for reduction of the fractured anatomy.These operative side images are processed to identify key structuresthat will be used for image registration with the nonoperative side.Additionally, these images may be corrected for distortion. 4) Thecomputing platform 100 identifies the best-matching nonoperative sideimage to the current operative side image using an image similaritymetric. The best-matching nonoperative side image is registered to thecurrent operative side image. The registration process computes an imagetransformation with is made of a transformation matrix (affine orrigid), a deformation grid, or both. 5) The computing platform 100 usesthe image transformation to align the non-operative-side image with thecurrent operative-side image and produce an image overlay thatillustrates the difference in the anatomical positioning of thenon-operative and operative-side images. This overlay image is aguidance pose-guide image 260 that is a template that the surgeon canuse to restore the patient's normal anatomy on the operative side (basedon the non-operative side anatomical positioning). 6) Any dimensionedgrids or other annotations placed on the non-operative side image can betransformed to their corresponding position on the operative-side imageto augment the registered composite image.

Now referring to FIG. 14A, a similarity evaluation 52 is performed andthe registration match confidence is calculated based in the ImageRegistration Module 9, which includes one or more image registrationalgorithms. The similarity evaluation 52 shows similarity evaluation andmapping.

In FIGS. 14 B-C, the grid similarity mapping of the confidencepercentage is shown in a graphical user interface display by aligningthe non-operative-side image with the current operative side. Arendering display of the ‘ghosting’ or good vs badmapping/matching/overlay of difference 53 that has been calculated bythe computing platform 100. The key color is the red showing the areadifference between the two images. Using the highlighted guide, thematch measurement 54 are shown to the user to accept or decline theinformation provided. In the user confirms good-side acceptable for usein for example alignment.

In FIGS. 15A-C, a graphical user interface displays the matchmeasurement 54 for the user to accept or decline the informationprovided. In an exemplary example, a fibula length match of −5.5 mm isshown. In FIG. 15 B, an example of a graphical user interface displayshows an ankle. The good side is over-laid with the grid alignment andmeasurements. A ghosting interpretation of an ankle 55 is shown. FIG.15C is an example of a hip graphical user interface demonstrating aghosting interpretation of a hip 56 on the good side overlay with gridalignment and measurements for a nail example.

In FIGS. 16 , the output of the Shape Modeling Module 10 is data forintraoperative calculation and guidance based on the 3D to 2Dregistration (or fitting) of a statistical shape model of theapplication anatomy to the 2D. A statistical shape model is used topredict the rotation in nailing applications. A statistical shape modelis a representation of the variability of anatomy in a population thatis encoded as one or more sample mean shapes plus modes of variability.A variety of image processing methods exist whereby a statistical shapemodel can be fit to images of patient anatomy in order to augmentmissing information in the medical image. For example, thethree-dimensional pose, including rotation, of a two-dimensionalfluoroscopic image of an implant such as an intertrochanteric nail canbe inferred using a statistical model of implant nail geometries. Thepose inference is computed by a simultaneous fitting of the statisticalmodel to the two-dimensional image and an iterative registration of thetwo-dimensional image with simulated projects of the statistical modelof the anatomy.

Now referring to FIGS. 17A&B for example, the computing platform 100identifies a varus reduction 60 of the femoral head during a procedureand provides the user with a varus reduction warning 61 indicating ahigh risk of failure based upon the calculations of the weightedintelligence model. A grid template of an optimal reduction can beprovided to provide guidance and assist the user in obtaining asatisfactory outcome.

The computing platform 100 is configured to analyze and interpret theinformation and provide guidance based upon a correlation to a known setof patterns and inference from datasets as set out in FIGS. 4A& 4B, theoutputs related to surgical guidance include implant selectionrecommendations, implant placement, performance predictions, probabilityof good outcomes, and failure risk scores. The structure of the deeplearning platform demonstrating the different layers is indicative ofthe configuration of the flow of information and how the information isapplied. The data layer is made of a collection of data from variousnetworks. In a trauma event, such as fracture reduction or deformitycorrection, or in an arthroplasty event such as hip or knee anatomicalalignment or bone cut guidance, or in the event of a spine procedurewith implant alignment correction, or in a sports medicine event withACL reconstruction alignment, these surgical and procedure specificdatasets coupled with domain knowledge and information from subjecthealth records that are critical to an event can be accessed. The dataset are used to interpret critical failure mode factors of an implant oranatomical alignment and combined will provide the user with a FailureRisk Score with the output to the user as a confidence percentagerecommendation of a suboptimal or optimal performance metric. The outputis presented to the user in the form of intelligent predictors andscores to support decisions encountered in a real time event.

The is computing platform 100 analyzes an image for risk factors thatthe user cannot see due to their human inability to interpret anoverwhelming amount of information at any specific moment. If theimplant placement and the alignment does not match this data pattern, itwill create an awareness in this specific situation and provide a hazardalert to the user. Essentially, identifying and predicting problemsahead of the user encountering them. This can lead to avoidance ofcomplications and prevention of errors. The surgical guidance is relatedto: deformity correction, an anatomy alignment, a fracture reduction andan anatomy reduction. The process of surgical guidance will be discussedin the following section for these applications. The following sectionsshow how the computing platform 100 interprets the information andprovides guidance based upon a correlation to a known set of patternsand inference from data sets as applied to different surgicalprocedures.

TRAUMA EXAMPLE—HIP FRACTURE. The most frequent fractures hospitalized inUS hospitals in 2015 were for those of the hip, according to data fromthe HCUP (Healthcare Cost and Utilization Project of the Agency forHealthcare Research and Quality (AHRQ)). There are known reasons for thefailure modes of these procedures. For example, it is documented thatdetermining and utilizing the correct entry point for nailing of a bonecan prevent malreduction of the fracture and ultimately failure of theimplant or the compromising of an optimal outcome.

Subject anatomy is unique and using a single-entry point for allsubjects is not desirable. Once the subject has been prepared forsurgery in a standard manner, the artificial intelligenceintra-operative surgical guidance system 1 is turned on and the platformis now activated. A new image of the subject's anatomy is taken. Theimage is of the contralateral unaffected, or good, side of the subject.The platform is instructed, or will determine, if the information itreceives is 3D or 2D. If the information is 3D, then the artificialintelligence intra-operative surgical guidance system 1 will call therelevant module to perform a 2D to 3D registration or utilize the 3Dmodel for statistical inference. If the image is 2D, the artificialintelligence intra-operative surgical guidance system 1 will capture theimage and an initial grid pose module will analyze the image anddetermine if the pose of the image is adequate for use as a ‘true’image. A true image is a datum or base image that will be utilizedthroughout the procedure as a good anatomical reference image that thesoftware algorithm is able to reproducibly recognize. With each newimage acquired during this step of the procedure, the algorithm willaccess a dataset of annotated good anatomy pose images and provide avirtual grid pose template to guide the user to establish the correctpose. An example of the grid pose in this situation would be to advisethe user to ‘internally rotate the hips 15 degrees in the AP image withthe beam centered at the teardrop’. Once the correct image is accepted,the computing platform 100 accessed the automated image segmentationalgorithm to identify the relevant anatomical features. In this specificapplication (FIG. 18A), the anatomical features to be identified caninclude the tip of the greater trochanter (GT) 70, femoral shaft axisidentified by center of canal 71, and the femoral neck axis identifiedby center of femoral head and center of neck 71. The neck-shaft angle 71is measured as the medial arc between the shaft and neck axes. Theaffected side image is now taken with a similar matching image pose. Thefeature recognition and segmentation algorithm are accessed as well. Theimage registration module 9 is accessed here and a good-side match modelis calculated. A good vs bad grid map similarity evaluation provides theuser with a match confidence percentage. This step involves: mapping agrid template to the anatomical structure to register an image for thenonoperative side of the subject's anatomy with an image of theintraoperative image of the operative side of the subject's anatomy toprovide a registered composite image. The registered composite image isprovided to the artificial intelligence engine to generate an at leastone graphical surgical guidance in this step, the implant dataset isaccessed. The dataset includes information on the three-dimensionalgeometry of the implant options. The machine learning algorithmassociated with this dataset analyzes, measures, and calculates therelevant variables and has the capability to identify suboptimal outputsand provide the user with situational awareness and hazard alertsleading to complication avoidance and error prevention. The output ispresented to the user as surgical guidance, such as an optimal orsub-optimal risk score for failure if the user proceeds with the pathwayalong which he intends follow.

The computing platform 100 predicts performance based upon thesepathways and can also provide the user with a probability of an optimalor sub-optimal outcome. The computing platform 100 provides the userwith an implant recommendation based upon the known predictors of asuccessful outcome. The computing platform 100 dynamically updates theregistered composite image with the at least one graphical surgicalguidance as the surgeon changes interoperative variables. The surgicalvariable depends upon the ongoing surgery and includes the position ofthe patient, the pose estimation, the implant position, or the nailentry point in a femur fracture surgical procedure.

Now referring to FIGS. 1A, 4A &. 18B, the correct nail entry-siteprediction information is now accessed within the known indicators andpredictors of complications dataset. The dataset uses information fromindustry gold-standard or historical peer-literature reviewed studies toperform analytical calculations and determine optimal and suboptimalperformance predictions. In addition, the computing platform 100determines if there is a probability of an error occurring by a specificplacement of an entry-site or orientation of a guidewire or reamer 75.The optimal entry point prediction utilizes a combination of the variousalgorithmic modules for information.

In this application, the relevant structures are identified using thesegmentation machine learning algorithm, the implant dataset is used forsubject best entry-point grid templating, the nail grid template isoptimally auto-placed on the image using the deep learning algorithmwith access to the known-literature and study dataset, the distancesbetween the various features are measured and the output values willpredict an outcome using the literature and study dataset with knownoptimal versus suboptimal risk values.

Now referring to FIG. 18C, for example, the user can want to use anentry-point at the GT (Greater Trochanter of Femur) tip 70, but theimplant design and subject's anatomy predicts this pathway will likelyresult in a valgus malreduction. Once an optimal entry point isquantified 76, it is displayed on all subsequent images acquired. Theentry point is dynamically tracked image to image by the anatomical andimplant segmentation tracking module. The updated entry point iscalculated relative to the known and selected anatomical featuretracking points.

Now referring to FIG. 18D, a new image is acquired and the guidewire 77or starter reamer is recognized on the image. An ideal system predictedentry point is recommended and displayed 76, and if the user acceptsthese suggestions, the user then places and orientates the guidewire orreamer in the positions guided by the accepted and displayed virtual oraugmented grid or avatar 78. The user completes this step of theprocedure with imaging or CAS (traditional Computer Assisted Surgerysystem) tracking and intelligence guidance from the systems' algorithmicmodules.

Now referring to FIG. 19A, this step is defined by the lag-screwplacement. The lag-screw module will calculate and provide guidance onthe depth, and rotation, of nail placement 80. Once the nail is begun tobe inserted, the lag-screw placement 81 will be used as the determinantof the depth the nail needs to be seated in the bone. When an image isacquired, the artificial intelligence intra-operative surgical guidancesystem 1 uses the hip application segmentation machine learning moduleto identify the relevant anatomical, instrument, and implant featuressuch as the nail alignment jig, and nail and lag-screw.

In addition, the artificial intelligence intra-operative surgicalguidance system 1 provides an automatic determination of screwtrajectories and more generally in situations of instrumentationtrajectories. For example, to determine if the instrumentation is withinthe right plane while simultaneously tracking anatomical, implant andinstrument considerations in different views. This is achieved usingdeep learning techniques, more specifically, a Reinforcement Learning(RL) technique. RL strategies are used to train an artificial RL agentto precisely and robustly identify/localize the optimal trajectory/pathby navigating in an environment, in our case the acquired fluoroscopicimages. The agent makes decisions upon which direction it has to proceedtowards an optimal path. By using such a decision-making process, the RLagent learns how to reach the final optimal trajectory. An RL agentlearns by interacting with an environment E. At every state (S), theregion of interest in this situation, a single decision is made tochoose an action (A), which consists of modifying the coordinates of thetrajectory, from a set of multiple discrete actions (A). Each validaction choice results in an associated scalar reward, defining thereward signal (R). The agent attempts to learn a policy to maximize bothimmediate and subsequent future rewards. The reward encourages the agentto move towards the best trajectory while still being learnable. Withthese considerations, we define the reward R=sgn(ED(Pi−1, Pt)—D(Pi,Pt)), where D is a function taking the Euclidean distance between planeparameters. Pi (Px, Py, Pz) is the current predicted trajectorycoordinate at step I and Pt is the target ground truth coordinates. Thedifference of the parameter distances, between the previous and currentsteps, signifies whether the agent is moving closer to or further awayfrom the desired plane parameters. Finally, the terminate state isreached once the RL agent reaches the target plane coordinates. The taskis to learn an optimal policy that maximizes the intermediate rewardsbut also to subsequent future rewards.

Now referring to FIG. 19 B, the nail is tracked in real-time usingsensors or images and is guided to the correct depth and the lag-screwconfiguration and placement information is provided as an input forinterpretation. The location of the lag-screw in the specific anatomy 82is analyzed to statistically determine its probability of cutting out ofthe femoral head 83.

Now referring to FIGS. 20A-C, the workflow of an artificial intelligenceassisted total hip arthroplasty (THA) is provided. In step 1, areference image is acquired and processed for a specific subject. Theworkflow to obtain a reference image of a subject depends on thehardware and anatomical features in the image of the subject. Sceneinterpretation is the process of applying automated artificialintelligence models, including a neural network model, wherein the oneor more automated artificial intelligence models are trained on aplurality of radiographic images from a data layer to automaticallyanalyze a radiographic image of a subject for relevant data. Sceneinterpretation can be used to generate the workflow.

Pre-Op Reference Image:

A radiographic image of the subject is acquired in the system (425). Ifno cup or stem, the process to acquire the correct reference imageinvolves detection of anatomical structures (known landmarks for aspecific procedure) and/or hardware in the pelvis/hip image. Theanteroposterior (AP) pelvis can be detected when these points areavailable: left and right teardrop, contra/ipsil and symphysis pubislandmark. For a contralateral approach: both sides of the subject areimaged. The required image is an anteroposterior hip view or ananteroposterior pelvis view. All structures that the system canrecognize need to be in the field of view. For an Ipsilateral approach:anatomical structures such as teardrop and symphysis pubis are detected.

Next at least one reference image metric is defined by the user. Themetrics can include: leg length and offset, ipsilateral, leg length andoffset, contralateral, pelvic tilt, pelvic rotation, femoral abduction(ipsil & contra), femoral rotation (ipsil & contra) and femoral headcenter of rotation (manual, if added). The step of landmark and hardwaredetection (428) is specific to what is identified from an anatomicallandmark, an instrument and/or an implant perspective.

The reference image can be a preoperative image that does not changeonce taken. Depending on the type of procedure, calibration of theimages of the subject may be required. The image is calibrated byidentification of a known shape and size of an object or anatomy withinthe image. For example, using the femoral head resection method, thestep of calibration includes manually entering a measuring of theresected femoral head diameter in the reference image when placingfemoral head calibration. Leg length offset is calibrated separatelyusing the contralateral comparison on the anteroposterior pelvis stemimage. A benefit of calibration is that quantitative leg length offsetmeasurements can be reported to the user to make intraoperativedecisions and pre-/post-op comparisons of images.

The next step involves evaluation of structures or hardware, such asimage identification in the pelvis/hip reference image (430). The levelanteroposterior (AP) pelvis can be detected when these points areavailable: both left and right teardrops and symphysis pubis structures.The system identifies metrics that include contralateral, pelvic tilt,pelvic rotation, femoral abduction contra only, femoral rotation contraonly and femoral head center of rotation (manual, if added). Landmarkdetection involves image interpretation. The landmark is detected by anobject detector model in real time by locating the landmark and classifyit with reference to the subject good side image. The step of imageidentification (430) is specific to then providing an output based uponthe step of landmark and hardware detection (428) findings and thenclassifying the image as an anteroposterior Hip/Cup (434) oranteroposterior Pelvis image (432).

The artificial intelligence intra-operative surgical guidance system ismade of a computer executing one or more automated artificialintelligence models trained on data layer datasets collections toidentify specific features or structures of anatomy or instruments andimplants (collectively hardware), and provide intra-operative surgicalguidance, and an output to a user, surgeon, or robot; and a displayconfigured to provide visual guidance to a user. The automatedartificial intelligence models include: Deep Learning, machine learningand reinforcement learning based techniques. For example, aConvolutional Neural Network (CNN) is trained using annotated labeledimages which include good and bad images to learn local image featureslinked to low-resolution, presence of noise/artifact, contrast/lightingconditions, etc. The Convolutional Neural Network model uses thelearning features to make feature or landmark identifications in a newimage. Module 7 provides an image annotation module that includes imageprocessing algorithms or advanced deep learning-based techniques fordetecting anatomical structures in a medical image and identifyingcontours or boundaries of anatomical objects in a medical image, such asbone or soft tissue boundaries.

Anatomical Landmark detection stands for the identification of keyelements of an anatomical body part that potentially have a high levelof similarity with the same anatomical body part of other patients. Thedeep learning algorithm encompasses various conventional layers, and itsfinal output layer provides self-driven data, including, but not limitedto, the system coordinates of important points in the image. In thecurrent invention, landmark detection can be also applied to determinesome key positions of anatomical parts in the body, for example,left/right of the femur, and left/right of the shoulder.

The deep neural network output is the annotated positions of theseanatomical parts. In this case, the deep learning algorithm uses atraining dataset which needs to meet some requirements: the firstlandmark in the first image used in the training must be consistentacross different images in the training dataset. Identifying contours ofanatomical objects refers to providing an edge map consisting of richhierarchical features of an image while preserving anatomical structureboundaries using deep learning techniques. A variety of highlyconfigurable deep learning architectures with an optimizedhyperparameters tuning are used to help with solving specific tasks.Calibration is optional for pre/post operative quantitative comparison.The image is calibrated by identification of a known shape and size ofan object or anatomy within the image.

Intra-Op Reaming for Cup Placement:

An intraoperative anteroposterior radiographic image is obtained can beeither anteroposterior hip or anteroposterior pelvis. Another step in anartificial intelligence assisted total hip arthroplasty (THA) is reaming(step 1.5). Hardware detection determines the workflow in the reamingoperation. The reamer is detected by the software and identified by itsshape and unique features such as the multiple holes in the instrument.If the femoral head center of rotation is defined, then structuresdetermine anteroposterior pelvis/hip view.

The next step in reaming involves determination of structures in thepelvis/hip image. A functional pelvis grid/template can be defined whenthese points are available: left/right teardrop, contra/ipsiILstructures, and symphysis pubis structures. In another alternativesurgical procedural process, anteroposterior hip (detected when limitedpoints available) involves detection of ipsilateral teardrop, ipsiIL andsymphysis pubis structures.

The next step in the reaming process involves comparing the referenceimage to the intraoperative images. The differences in the imagesprovide: visual reference of defined center or rotation (contra or ipsibased on availability), and provides difference from reference imagefor: the pelvic tilt, pelvic rotation, contralateral femoralrotation/abduction (availability based on ref image type). Thedifferences in the images provide a visual reference of ipsilateralcenter of rotation, difference from reference image for: pelvic tilt andpelvic rotation. In the instance when center of rotation not defined.

The next step in reaming involves determination of structures in thepelvis/hip image. The anteroposterior pelvis can be detected when thesepoints are available: left/right teardrop, contra/ipsiILT and symphysispubis structures.

The next step in the reaming process involves comparing the referenceimage to the intraoperative images. The differences in the images lookat: pelvic tilt, pelvic rotation, and contralateral femoralrotation/abduction in an alternative embodiment, taken from theipsilateral side: anteroposterior Hip (detected when limited pointsavailable). These points include: ipsilateral teardrop, ipsiILT andsymphysis pubis structures. The next step in the reaming processinvolves comparing the reference image to the intraoperative images. Thedifferences in the images look at images for: pelvic tilt and pelvicrotation.

INTRA-OP CUP PLACEMENT Step 2 is cup placement. An intraoperativeanteroposterior cup radiographic image is obtained (434). Hardwaredetection determines the workflow in the cup placement operation. If acup is detected and no ipsilateral stem, or no reamer is detected, thencup placement procedure proceeds. The next step in cup placementinvolves determination of structures in the pelvis/hip image. Ananteroposterior pelvis image is detected when these points areavailable: Various detections contra/ipsilateral structures, such asleft/right teardrop and symphysis pubis structures. The next step is cupvirtual ellipse grid/template calculation and generation followed byautomatically snapping (registering) specific points on the virtual gridtemplate to the known matching relevant anatomical landmark or implantpoints. The implanted cup in this situation is positioned in the anatomyand the sequence of images allow for identification of the cup by thesoftware and subsequently identifying the opening of the cup which hasthe shape of an ellipse. The major and minor axes of the cup arecalculated, and the axes end points are quantified on the image. Nextcup measurements are reported: the cup opening, and inclination andversion angle measurements are provided. The image can be calibratedbased on the cup if other measurements are required.

In an alternative embodiment, if only the ipsilateral side is imagedthen, an anteroposterior hip workflow is intended (detected when limitedpoints available), these points include: detection of ipsilateralteardrop; ipsiIL and symphysis pubis structures. Difference is comparedto the reference image including: pelvic tilt and pelvic rotation. Nextcup measurements are reported: Cup ellipse registers to cup opening andinclination and version measurements are provided (image can becalibrated based on cup if other measurements are desired).

INTRA-OP STEM PLACEMENT and LLO GUIDANCE. An intraoperativeanteroposterior pelvis radiographic image is obtained (432). The nextstep, step 3, is stem placement. Hardware detection determines theworkflow in the stem placement operation. If a cup and a stem isdetected for example, or a reamer is detected, then stem placementprocedure proceeds. The next step in stem placement involvesdetermination of structures in the pelvis/hip image, for exampleanteroposterior pelvis (detected when these points are available)left/right teardrop, contra/ipsiILT and symphysis pubis structures. Thenext step in the stem placement process involves comparing the referenceimage to the intraoperative images. The differences in the images lookat: difference from reference image for: pelvic tilt, pelvic rotation,contralateral and ipsilateral femoral rotation/abduction.

In the next step of the process, calibration is performed, and leglength and offset measurements are reported to the user. The image maybe calibrated by autodetection of objects, anatomy, instruments orimplants, and a manual value option may also be entered, and the leglength and offset are reported in mm (+/−2 mm). Additionally,calibration object (detected or selected) with manual value entered leglength and offset reported in mm (+/−2 mm). If the image is notcalibrated, then leg length and offset is displayed as a visual gridfrom the reference image.

In an alternative embodiment, the next step in cup placement involvesdetermination of structures in the pelvis/hip image. An anteroposteriorpelvis is detected when these points are available: anteroposterior hip(detected when limited points available) Detects ipsilateral teardrop,ipsiILT, symphysis pubis structures, Ipsilateral approach only needsipsilateral image and Contralateral approach needs ipsilateral andcontralateral approach needs ipsilateral and contralateral. If the imageis calibrated, then autodetection of cup opening or cup diameter withmanual value entered leg length and offset reported in mm (+/−2 mm) or acalibration object (detected or selected) with manual value entered leglength and offset reported in mm (+/−2 mm). If the image is notcalibrated, then leg length and offset is displayed as a visual gridfrom the reference image.

REGISTRATION: The process of interoperative registration of agrid/template can be automated by image interpretation and smart gridlogic. More specifically, for example, if a cup reamer or cup implantsare detected in the image, then the system registers a cup grid to thesestructures: left teardrop and right teardrop and major axis of thepoints of the cup. A grid/template is a graphical representation of datagenerated based upon identified and relevant registration to anatomicalstructures. The grid/template provides a visual presentation in a mannerthat provides the user with measures for implant placement and guidance.In this specific case, a cup grid is a grid that is generated based uponanatomical structures tear drop, symphysis pubis and the cup implantthat is specific for a patient. Once the grid/template is registered tothe structure in the intraoperative image, then then geometry ismeasured and data is outputted (for example for Cup Abduction, CupVersion, such as Fit ellipse to cup for version angle), and measureangles relative to True Pelvis Grid (tear drops and symphysis pubistriangle). The system provides situational awareness and warning ifnumbers outside of safe zones or acceptable parameters by applyingModule 13.

Now referring to FIG. 20C, structures are detected by the system inevery image that is taken of the subject. The system is trained tosearch for specific structures such as symphysis pubis (SP's), teardrop(TD's), Lesser Trochanter (LT's) etc. The system determines if a subjectreference grid has been previously saved (450). If not, a functionalpelvis grid/template 200 is generated by the system. A functional pelvisgrid/template 200 is a subject specific grid that is specificallygenerated by the system based upon known specific structures of theanatomy it is identifying, such as teardrops and symphysis pubis. Thesystem is configured to automatically identify anatomical structures,instruments and implants and then based upon logic and sceneinterpretation; the system identifies the relevant structures needed fora specific virtual grid for a specific image scene.

Now referring to FIGS. 28-32 , a situation specific grid can begenerated by the computing platform. These situation specific gridsinclude: functional pelvis grid, spinopelvic grid, level pelvis grid,reference grid, neck cut grid, reamer depth grid, center of rotationgrid, cup grid, leg length and offset grid, and femur abduction grid. Ifa functional pelvis grid/template is saved then, then the systemdetermines if two tear drops and an ipsilateral teardrop are detected inthe image 452. If detected, then an anteroposterior pelvis view of thesubject is received into the system. If various structures can bedetected in these views, including for example, two teardrops,ipsilateral teardrop, ipsiIL and symphysis pubis, then the computingplatform determines if any hardware is detected in the image (454). Ifthe system detects hardware in the anteroposterior pelvis image, thenthe system determines if the anteroposterior pelvis image has a stempresent on the operative side (456). If no hardware is detected in theanteroposterior pelvis image of the subject, then the functional pelvisgrid/template is registered to the intraoperative image. If a stem isdetected in the anteroposterior pelvis image on the operative side, thenthe leg length offset grid and the functional pelvis grid/template areregistered to the intraoperative image, If the subject does not have astem on the operative side, cup (ellipse tool and reference grid) isregistered to the intraoperative image. If the cup is not centered, thenthe functional pelvis grid/template angle is locked. The computingplatform determines if one tear drop is detected (458). If yes, then ananteroposterior hip image of the subject is received into the system.This image is evaluated to determine if any of these structures arepresent: teardrop, symphysis pubis and one side ipsilateral is detected.If these structures are detected, then the computing platform determinesif hardware (460) is present in the image. The computing platformdetermines if a stem is present in the image (462). If a stem isdetected in the image, the leg length and off set grid and the referencegrid are registered to the intraoperative image. If no stem is detectedin the image, the cup (ellipse tool and the reference grid). If nohardware present in the image, the computing platform registers thefunctional pelvis grid/template to the intraoperative image.

Now referring to FIGS. 21A-B, an artificial intelligence assisted totalhip arthroplasty (THA) image is shown. The actions required are togenerate a subject specific functional pelvis grid template 200 and ananatomy map 201 (FIG. 21A), The anatomy map 201 (as shown in FIG. 21A)is a map of anatomical structures i.e., landmarks used in the proceduresuch as Lesser Trochanter points 05, 06 and 07 left (L) and right (R)used for leg length and offset determination. For example, to determineleg length using the generated anatomy map, the steps include: autoidentification of all the relevant individual anatomical structures i.e.landmarks (such as Teardrop, Symphysis Pubis) and then use thesemultitude of points to generate a map of the entire anatomical region(for example pelvis, hip), which are then used to quantify measurementsbetween these various points/structures i.e. landmarks, and then basedupon the output required (in this instance leg length), the vector orvertical distances measured from specific map landmarks or coordinatessuch as the Teardrop and Lesser Trochanter will be a value used todefine leg length. Other landmarks, such as center of femoral head totrans teardrop line are also used to measure leg length. The stepsinvolved in using an anatomy map to determine Offset include: autoidentification of all the relevant individual anatomical landmarks (suchas Teardrop, Lesser Trochanter) and then use these multitude of pointsto generate a map of the entire anatomical region (for example pelvis,hip), which are then used to quantify measurements between these variouspoints/landmarks, and then based upon the output required (in this caseOffset), the vector or horizontal distances measured from specific maplandmarks or coordinates such as the Teardrop and Lesser Trochanter willbe a value used to define Offset. Other landmarks, such as center offemoral head to trans teardrop line are also used to measure Offset.

For grid generation, this is accomplished by the steps of: identifyingthe left teardrop (TD) 01L; identifying the right teardrop (TD) 01R,identifying the symphysis pubis (SP) points 02 and 03, or other relevantfeatures such as numbered 1-13 L&R; calibrating the reference image;drawing and measuring the trans-teardrop lines (TTD); drawing andmeasuring the teardrop (TD) 01 to symphysis pubis (SP) 02 line anddrawing and measuring the vertical and horizontal distances from thevarious structures. The output of this is: a functional pelvis grid ortemplate 200 (as shown in FIG. 21B). The functional pelvis grid 200 is asubject specific grid for the specific purpose of defining a true orlevel functional pelvis as a reference guide for the entireprocedure/surgery.

Now referring to FIG. 22 the intra-operative image is calibrated. Inthese images there is no stem or cup, but an instrument in the field ofview, for example a caliper which is recognized by the system as anobject used for calibration such as caliper, ball, cup, or anatomy. Theoutput of the scaling process is the scale of the image. Calibration isachieved by the system identifying a known object 1000 and shape andsize and based upon that known measurement apply dimensions to theanatomy data map. In this image hardware 1010 is shown.

Now referring to FIGS. 23A-C an intraoperative image is shown of ananteroposterior pelvis with instruments and an implant: for example,Hohmann retractor, a cup impactor with an implant cup (with no stem)1001 are shown. The action is registering the cup grid 1005 to theanatomical structures 1000 (FIG. 23A) and fitting an ellipse 1003 (FIG.23C) to the cup 1001 (FIG. 23B). The ellipse 1003 is the outline of theedge of the cup 1001 when viewed at an angle such as from a fluoroscopicimage pose. The output is a cup inclination and cup version numbersshown in FIG. 23B. Cup version is the anteversion angle of a cup (shortaxis opening/measure of the ellipse) relative to a pelvis in theanteroposterior image view. This angle is displayed as a cup versionangle number. Here a hidden virtual functional pelvis grid or templateand the implant (cup) 1001 is shown in FIG. 23C. In FIG. 23C, thegraphical user interface shows a cup grid 1002 and a cup ellipse 1003.

Now referring to FIG. 24 in this intraoperative image, theanteroposterior pelvis implant cup 1001 and stem (trial) 1004 are shown.The cup opening (ellipse) 1003 is shown, the action involves: providinga functional pelvis grid or template 1005 and then registering: a leglength (LL) and offset (OFF) grid 1005 to the anatomical structures, forexample Teardrops and Lesser Trochanters to the leg length and off setgrid 1005 is related to the length and offset of the leg. The output isleg length and offset data. The leg and off set grid 1005 grid isregistered 1012 good side over bad side.

Now referring to FIGS. 25A-B, an intraoperative anteroposterior image ofa hip with no cup or stem is shown. The action, if one ipsilateralteardrop plus one femoral head, plus one teardrop and no implants areidentified in the image scene, then identify the anatomical structuresand register a femur abduction grid/template to the anatomicalstructures or acquisition of a good side image for use in the overlay orregistration to the bad side image. The output is an image of the femurshowing the overlay (FIG. 26A). An intraoperative image of ananteroposterior pelvis is shown along with an implant cup and stem (FIG.26 B). Here the good side is registered to the bad side and thatdifferences can be quantified between the two images (FIG. 26A). Theaction involves the identification of one ipsilateral teardrop 1 plusone other anatomical landmark 9-25 plus one Lesser Trochanter 11 andimplants. This image is for registration/overlay of good side over badside. The output of the registration of good side and bad side can beshown in FIG. 26A. The virtual position of the good/contralateral sideLesser Trochanter, which the target position of the bad/affected sidehip length. The output is leg length and offset data.

Now referring to FIGS. 27A-B, if the computing platform detectsteardrops, symphysis pubis, ipsilateral teardrop, implant (cup) and stem(implant) and femoral head implant, then a functional pelvisgrid/template 200 is registered to the image. If the cup Implant andstem implant are in the image, then the computing platform registers theleg length (LL) and offset (OFF) grid 200 to provide Leg Length & Offsetdata. The data output used by the surgeon to make implant decisions.

The foregoing detailed description has been presented for purposes ofillustration and description. It is not intended to be exhaustive or tolimit the invention to the precise form disclosed. Many modificationsand variations are possible considering the above teachings. Thedescribed embodiments were chosen to best explain the principlesinvolved and their practical applications, to thereby enable othersskilled in the art to best utilize the various embodiments and withvarious modifications as are suited to the use contemplated. It isintended that the scope of the invention be defined by the claimsappended here.

We claim: 1.-8. (canceled)
 9. An artificial intelligence assisted totalhip arthroplasty comprising: providing a computing platform comprised ofan at least one image processing algorithm for the classification of aplurality of intra-operative medical images, the computing platformconfigured to execute one or more automated artificial intelligencemodels, wherein the one or more automated artificial intelligence modelscomprises a neural network model, wherein the one or more automatedartificial intelligence models are trained on data from a data layer toidentify a plurality of anatomical structures and a plurality ofhardware; receiving a preoperative radiographic image of a subject;detecting the plurality of anatomical structures in the preoperativeradiographic image of the subject; generating a subject specificfunctional pelvis grid from the plurality of anatomical structuresdetected in the preoperative radiographic image of the subject,receiving an intraoperative anteroposterior pelvis radiographic image ofthe subject; identifying an object selected from the group consistingof: an at least one anatomical landmark and an at least one hardware inthe intraoperative anteroposterior pelvis radiographic image, wherebythe computing platform performs the step of: selecting a situationspecific grid selected from the group consisting of: functional pelvisgrid, spinopelvic grid, level pelvis grid, reference grid, neck cutgrid, reamer depth grid, center of rotation grid, cup grid, leg lengthand offset grid, and femur abduction grid; and registering the selectedgrid to the anatomical landmark in the intraoperative image.
 10. Themethod of claim 9, comprising the step of registering the subject goodside radiographic image to the intraoperative tide radiographic image.11. The method of claim 9, wherein the data layer includes at least aplurality of fluoroscopic surgical images, wherein the automatedartificial intelligence models are trained to calculate intra-operativesurgical decision risks, further comprising the step of: arranging thegraphical representation within a visual display to provide situationalawareness to a user as a function of the classification of surgicaloutcomes.
 12. The method of claim 9, further comprising the step of:constructing a graphical representation of data, wherein the graphicalrepresentation is an anatomy map of an entire anatomical region; theanatomy map generated based upon the anatomical landmark detected by thecomputing platform. 13.-15. (canceled)